chore: add more accuracy models via Grove MCP-491#1140
Conversation
There was a problem hiding this comment.
Pull request overview
This PR updates the accuracy test runner to use Grove-backed LLM models (plus Anthropic via Grove), adds a model allowlist mechanism, and adjusts accuracy test infrastructure/scripts to better support local and CI runs.
Changes:
- Replace direct OpenAI/Gemini accuracy models with multiple Grove-backed models and add optional
MDB_ACCURACY_MODEL_ALLOWLISTfiltering. - Adjust the accuracy test client process environment handling and disk result status updates.
- Update scripts/workflows/editor config to support the new providers and higher test concurrency.
Reviewed changes
Copilot reviewed 8 out of 9 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/accuracy/sdk/models.ts | Switches to Grove-backed model providers, adds allowlist-based model filtering. |
| tests/accuracy/sdk/accuracyTestingClient.ts | Strips MDB_MCP_* env vars when spawning the test server process. |
| tests/accuracy/sdk/accuracyResultStorage/diskStorage.ts | Avoids updating run status when the results file doesn’t exist. |
| tests/accuracy/createDeployment.test.ts | Loosens expected matcher for imageTag parameter. |
| scripts/accuracy/runAccuracyTests.sh | Documents Grove env vars/allowlist and increases Vitest worker count. |
| package.json | Adds @ai-sdk/anthropic dev dependency. |
| pnpm-lock.yaml | Locks @ai-sdk/anthropic and related transitive dependencies. |
| .vscode/launch.json | Adds a VS Code launch config for debugging accuracy tests. |
| .github/workflows/accuracy-tests.yml | Switches CI accuracy runs to use MDB_GROVE_API_KEY instead of OpenAI/Gemini keys. |
Files not reviewed (1)
- pnpm-lock.yaml: Language not supported
Coverage Report for CI Build 25323040638Warning No base build found for commit Coverage: 81.733%Details
Uncovered ChangesNo uncovered changes found. Coverage RegressionsRequires a base build to compare against. How to fix this → Coverage Stats
💛 - Coveralls |
nirinchev
changed the title
|
@copilot resolve the merge conflicts in this pull request |
|
This PR has gone 30 days without any activity and meets the project's definition of "stale". This will be auto-closed if there is no new activity over the next 30 days. If the issue is still relevant and active, you can simply comment with a "bump" to keep it open, or add the label "not_stale". Thanks for keeping our repository healthy! |
Proposed changes
This adds some Grove-backed models for the accuracy test runner to use.